{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "FinetuneCTC.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vqepaR4bZzJ"
      },
      "source": [
        "# Tutorial on ASR Finetuning with CTC model \n",
        "Let's finetune a pretrained ASR model!\n",
        "\n",
        "Here we provide pre-trained speech recognition model with CTC loss that is trained on many open-sourced datasets. Details can be found in [Rethinking Evaluation in ASR: Are Our Models Robust Enough?](https://arxiv.org/abs/2010.11745)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AbTYA-j_vk-7"
      },
      "source": [
        "## Step 1: Install `Flashlight`\n",
        "First we install `Flashlight` and its dependencies. Flashlight is built from source with either CPU/CUDA backend and installation takes **~16 minutes**. \n",
        "\n",
        "For installation out of colab notebook please use [link](https://github.com/fairinternal/flashlight#building). \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oDJo0P8DRL3N"
      },
      "source": [
        "# First, choose backend to build with\n",
        "backend = 'CUDA' #@param [\"CPU\", \"CUDA\"]\n",
        "# Clone Flashlight\n",
        "!git clone https://github.com/flashlight/flashlight.git\n",
        "# install all dependencies for colab notebook\n",
        "!source flashlight/scripts/colab/colab_install_deps.sh\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yXKe3wUWiWV"
      },
      "source": [
        "Build CPU/CUDA Backend of `Flashlight`:\n",
        "- Build from current master. \n",
        "- Builds the ASR app. \n",
        "- Resulting binaries in `/content/flashlight/build/bin/asr`.\n",
        "\n",
        "If using a GPU Colab runtime, build the CUDA backend; else build the CPU backend."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x5QFOur3AcB1"
      },
      "source": [
        "# export necessary env variables\n",
        "%env MKLROOT=/opt/intel/mkl\n",
        "%env ArrayFire_DIR=/opt/arrayfire/share/ArrayFire/cmake\n",
        "%env DNNL_DIR=/opt/dnnl/dnnl_lnx_2.0.0_cpu_iomp/lib/cmake/dnnl\n",
        "\n",
        "if backend == \"CUDA\":\n",
        "  # Total time: ~13 minutes\n",
        "  !cd flashlight && git checkout d2e1924cb2a2b32b48cc326bb7e332ca3ea54f67 && mkdir -p build && cd build && \\\n",
        "  cmake .. -DCMAKE_BUILD_TYPE=Release \\\n",
        "           -DFL_BUILD_TESTS=OFF \\\n",
        "           -DFL_BUILD_EXAMPLES=OFF \\\n",
        "           -DFL_BUILD_APP_ASR=ON && \\\n",
        "  make -j$(nproc)\n",
        "elif backend == \"CPU\":\n",
        "  # Total time: ~14 minutes\n",
        "  !cd flashlight && git checkout d2e1924cb2a2b32b48cc326bb7e332ca3ea54f67 &&  mkdir -p build && cd build && \\\n",
        "  cmake .. -DFL_ARRAYFIRE_USE_CPU=ON \\\n",
        "           -DFL_USE_ARRAYFIRE=ON \\\n",
        "           -DFL_BUILD_TESTS=OFF \\\n",
        "           -DFL_BUILD_EXAMPLES=OFF \\\n",
        "           -DFL_BUILD_APP_ASR=ON && \\\n",
        "  make -j$(nproc)\n",
        "else:\n",
        "  raise ValueError(f\"Unknown backend {backend}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-V0U-Dow-vs"
      },
      "source": [
        "Let's take a look around."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k6tRnX1iHCoX",
        "outputId": "e7a0fb5f-3507-4c7e-801c-9308ea65637e"
      },
      "source": [
        "# Binaries are located in\n",
        "!ls flashlight/build/bin/asr"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fl_asr_align   fl_asr_tutorial_finetune_ctc\n",
            "fl_asr_decode  fl_asr_tutorial_inference_ctc\n",
            "fl_asr_test    fl_asr_voice_activity_detection_ctc\n",
            "fl_asr_train\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__56IY0xcgMv"
      },
      "source": [
        "## Step 2: Setup Finetuning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T9078zTzcdS3"
      },
      "source": [
        "#### Downloading the model files\n",
        "\n",
        "First, let's download the pretrained models for finetuning. \n",
        "\n",
        "For acoustic model, you can choose from \n",
        "\n",
        ">Architecture | # Params | Criterion | Model Name | Arch Name \n",
        ">---|---|:---|:---:|:---:\n",
        "> Transformer|70Mil|CTC|am_transformer_ctc_stride3_letters_70Mparams.bin |am_transformer_ctc_stride3_letters_70Mparams.arch\n",
        "> Transformer|300Mil|CTC|am_transformer_ctc_stride3_letters_300Mparams.bin | am_transformer_ctc_stride3_letters_300Mparams.arch\n",
        "> Conformer|25Mil|CTC|am_conformer_ctc_stride3_letters_25Mparams.bin|am_conformer_ctc_stride3_letters_25Mparams.arch\n",
        "> Conformer|87Mil|CTC|am_conformer_ctc_stride3_letters_87Mparams.bin|am_conformer_ctc_stride3_letters_87Mparams.arch\n",
        "> Conformer|300Mil|CTC|am_conformer_ctc_stride3_letters_300Mparams.bin| am_conformer_ctc_stride3_letters_300Mparams.arch\n",
        "\n",
        "For demonstration, we will use the model in first row and download the model and its arch file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xn8XH01Chilw"
      },
      "source": [
        "!wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/am_transformer_ctc_stride3_letters_70Mparams.bin -O model.bin # acoustic model\n",
        "!wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/am_transformer_ctc_stride3_letters_70Mparams.arch -O arch.txt # model architecture file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlArtmiEhitM"
      },
      "source": [
        "Along with the acoustic model, we will also download the tokens file, lexicon file "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RPDEovLVc46Z"
      },
      "source": [
        "!wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/tokens.txt -O tokens.txt # tokens (defines predicted tokens)\n",
        "!wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/lexicon.txt -O lexicon.txt #  lexicon files (defines mapping between words)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UKZpiqt9bM4a"
      },
      "source": [
        "#### Downloading the dataset\n",
        "\n",
        "For finetuning the model, we provide a limited supervision dataset based on [AMI Corpus](http://groups.inf.ed.ac.uk/ami/corpus/). It consists of 10m, 1hr and 10hr subsets organized as follows. \n",
        "\n",
        "```\n",
        "dev.lst           # development set \n",
        "test.lst          # test set \n",
        "train_10min_0.lst # first 10 min fold\n",
        "train_10min_1.lst\n",
        "train_10min_2.lst\n",
        "train_10min_3.lst\n",
        "train_10min_4.lst\n",
        "train_10min_5.lst\n",
        "train_9hr.lst     # remaining data of the 10h split (10h=1h+9h)\n",
        "```\n",
        "The 10h split is created by combining the data from the 9h split and the 1h split. The 1h split is itself made of 6 folds of 10 min splits.\n",
        "\n",
        "The recipe used for preparing this corpus can be found [here](https://github.com/flashlight/wav2letter/tree/master/data/ami). \n",
        "\n",
        "**You can also use your own dataset to finetune the model instead of AMI Corpus.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "naYUr0cOjF1Q",
        "outputId": "cb9ff351-a9ad-4232-d4ab-d72f34796098"
      },
      "source": [
        "!rm ami_limited_supervision.tar.gz \n",
        "!wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/ami_limited_supervision.tar.gz -O ami_limited_supervision.tar.gz\n",
        "!tar -xf ami_limited_supervision.tar.gz \n",
        "!ls ami_limited_supervision"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "rm: cannot remove 'ami_limited_supervision.tar.gz': No such file or directory\n",
            "audio\t  train_10min_0.lst  train_10min_3.lst\ttrain_9hr.lst\n",
            "dev.lst   train_10min_1.lst  train_10min_4.lst\n",
            "test.lst  train_10min_2.lst  train_10min_5.lst\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mj32IH3HvjZo"
      },
      "source": [
        "### Get baseline WER before finetuning\n",
        "\n",
        "Before proceeding to finetuning, let's test (viterbi) WER on AMI dataset to we have something to compare results after finetuning."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ppGOZZWR4t_y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ecfb6def-2e4f-4482-ddc4-a710e6e5cc5f"
      },
      "source": [
        "! ./flashlight/build/bin/asr/fl_asr_test --am model.bin --datadir '' --emission_dir '' --uselexicon false \\\n",
        "            --test ami_limited_supervision/test.lst --tokens tokens.txt --lexicon lexicon.txt --show "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[1;30;43mStreaming output truncated to the last 100 lines.\u001b[0m\n",
            "[sample: EN2002b_H02_FEO072_4.48_4.59, WER: 100%, TER: 100%, total WER: 26.5703%, total TER: 13.0473%, progress (thread 0): 99.7389%]\n",
            "|T|: y e p\n",
            "|P|: \n",
            "[sample: EN2002b_H03_MEE073_307.97_308.08, WER: 100%, TER: 100%, total WER: 26.5712%, total TER: 13.0479%, progress (thread 0): 99.7468%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002b_H03_MEE073_1578.82_1578.93, WER: 100%, TER: 100%, total WER: 26.572%, total TER: 13.0485%, progress (thread 0): 99.7547%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002c_H03_MEE073_1446.36_1446.47, WER: 100%, TER: 100%, total WER: 26.5728%, total TER: 13.0491%, progress (thread 0): 99.7627%]\n",
            "|T|: o h\n",
            "|P|: \n",
            "[sample: EN2002d_H01_FEO072_118.11_118.22, WER: 100%, TER: 100%, total WER: 26.5736%, total TER: 13.0495%, progress (thread 0): 99.7706%]\n",
            "|T|: s o\n",
            "|P|: \n",
            "[sample: EN2002d_H02_MEE071_2107.05_2107.16, WER: 100%, TER: 100%, total WER: 26.5744%, total TER: 13.0499%, progress (thread 0): 99.7785%]\n",
            "|T|: t\n",
            "|P|: \n",
            "[sample: ES2004c_H02_MEE014_1184.81_1184.91, WER: 100%, TER: 100%, total WER: 26.5753%, total TER: 13.0501%, progress (thread 0): 99.7864%]\n",
            "|T|: y e s\n",
            "|P|: \n",
            "[sample: IS1009d_H01_FIO087_1744.56_1744.66, WER: 100%, TER: 100%, total WER: 26.5761%, total TER: 13.0507%, progress (thread 0): 99.7943%]\n",
            "|T|: o h\n",
            "|P|: \n",
            "[sample: TS3003c_H02_MTD0010ID_1023.66_1023.76, WER: 100%, TER: 100%, total WER: 26.5769%, total TER: 13.0512%, progress (thread 0): 99.8022%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002a_H00_MEE073_2013_2013.1, WER: 100%, TER: 100%, total WER: 26.5777%, total TER: 13.0518%, progress (thread 0): 99.8101%]\n",
            "|T|: y e a h\n",
            "|P|: \n",
            "[sample: EN2002b_H00_FEO070_43.63_43.73, WER: 100%, TER: 100%, total WER: 26.5785%, total TER: 13.0526%, progress (thread 0): 99.818%]\n",
            "|T|: y e a h\n",
            "|P|: \n",
            "[sample: EN2002b_H03_MEE073_293.52_293.62, WER: 100%, TER: 100%, total WER: 26.5794%, total TER: 13.0534%, progress (thread 0): 99.826%]\n",
            "|T|: y e p\n",
            "|P|: e\n",
            "[sample: EN2002c_H03_MEE073_763_763.1, WER: 100%, TER: 66.6667%, total WER: 26.5802%, total TER: 13.0537%, progress (thread 0): 99.8339%]\n",
            "|T|: m m\n",
            "|P|: \n",
            "[sample: EN2002d_H03_MEE073_2099.51_2099.61, WER: 100%, TER: 100%, total WER: 26.581%, total TER: 13.0541%, progress (thread 0): 99.8418%]\n",
            "|T|: ' k a y\n",
            "|P|: \n",
            "[sample: ES2004a_H00_MEO015_168.79_168.88, WER: 100%, TER: 100%, total WER: 26.5818%, total TER: 13.055%, progress (thread 0): 99.8497%]\n",
            "|T|: y e p\n",
            "|P|: \n",
            "[sample: ES2004c_H00_MEO015_360.73_360.82, WER: 100%, TER: 100%, total WER: 26.5826%, total TER: 13.0556%, progress (thread 0): 99.8576%]\n",
            "|T|: m m h m m\n",
            "|P|: \n",
            "[sample: ES2004c_H03_FEE016_1132.2_1132.29, WER: 100%, TER: 100%, total WER: 26.5835%, total TER: 13.0566%, progress (thread 0): 99.8655%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: ES2004c_H03_FEE016_1651.84_1651.93, WER: 100%, TER: 100%, total WER: 26.5843%, total TER: 13.0572%, progress (thread 0): 99.8734%]\n",
            "|T|: m m h m m\n",
            "|P|: \n",
            "[sample: IS1009c_H02_FIO084_1753.49_1753.58, WER: 100%, TER: 100%, total WER: 26.5851%, total TER: 13.0582%, progress (thread 0): 99.8813%]\n",
            "|T|: y e s\n",
            "|P|: \n",
            "[sample: IS1009d_H01_FIO087_749.88_749.97, WER: 100%, TER: 100%, total WER: 26.5859%, total TER: 13.0588%, progress (thread 0): 99.8892%]\n",
            "|T|: m m h m m\n",
            "|P|: \n",
            "[sample: IS1009d_H02_FIO084_1718.32_1718.41, WER: 100%, TER: 100%, total WER: 26.5867%, total TER: 13.0598%, progress (thread 0): 99.8972%]\n",
            "|T|: w h a t\n",
            "|P|: \n",
            "[sample: TS3003d_H00_MTD009PM_1597.33_1597.42, WER: 100%, TER: 100%, total WER: 26.5875%, total TER: 13.0606%, progress (thread 0): 99.9051%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002a_H00_MEE073_887.36_887.45, WER: 100%, TER: 100%, total WER: 26.5884%, total TER: 13.0612%, progress (thread 0): 99.913%]\n",
            "|T|: d a m n\n",
            "|P|: \n",
            "[sample: EN2002a_H00_MEE073_1500.91_1501, WER: 100%, TER: 100%, total WER: 26.5892%, total TER: 13.062%, progress (thread 0): 99.9209%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002c_H03_MEE073_429.63_429.72, WER: 100%, TER: 100%, total WER: 26.59%, total TER: 13.0626%, progress (thread 0): 99.9288%]\n",
            "|T|: y e a h\n",
            "|P|: a\n",
            "[sample: EN2002c_H02_MEE071_2578.94_2579.03, WER: 100%, TER: 75%, total WER: 26.5908%, total TER: 13.0632%, progress (thread 0): 99.9367%]\n",
            "|T|: d a m n\n",
            "|P|: \n",
            "[sample: EN2002d_H03_MEE073_999.94_1000.03, WER: 100%, TER: 100%, total WER: 26.5916%, total TER: 13.064%, progress (thread 0): 99.9446%]\n",
            "|T|: m m\n",
            "|P|: \n",
            "[sample: ES2004a_H00_MEO015_252.48_252.56, WER: 100%, TER: 100%, total WER: 26.5925%, total TER: 13.0644%, progress (thread 0): 99.9525%]\n",
            "|T|: m m\n",
            "|P|: \n",
            "[sample: ES2004c_H00_MEO015_1885.03_1885.11, WER: 100%, TER: 100%, total WER: 26.5933%, total TER: 13.0648%, progress (thread 0): 99.9604%]\n",
            "|T|: o h\n",
            "|P|: \n",
            "[sample: TS3003a_H01_MTD011UID_1313.86_1313.94, WER: 100%, TER: 100%, total WER: 26.5941%, total TER: 13.0652%, progress (thread 0): 99.9684%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002a_H00_MEE073_185.9_185.98, WER: 100%, TER: 100%, total WER: 26.5949%, total TER: 13.0658%, progress (thread 0): 99.9763%]\n",
            "|T|: g\n",
            "|P|: \n",
            "[sample: EN2002d_H03_MEE073_241.62_241.69, WER: 100%, TER: 100%, total WER: 26.5957%, total TER: 13.066%, progress (thread 0): 99.9842%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: IS1009a_H02_FIO084_490.4_490.46, WER: 100%, TER: 100%, total WER: 26.5966%, total TER: 13.0666%, progress (thread 0): 99.9921%]\n",
            "|T|: ' k a y\n",
            "|P|: \n",
            "[sample: EN2002b_H03_MEE073_613.94_614, WER: 100%, TER: 100%, total WER: 26.5974%, total TER: 13.0674%, progress (thread 0): 100%]\n",
            "I1224 05:05:23.029392  9187 Test.cpp:418] ------\n",
            "I1224 05:05:23.029417  9187 Test.cpp:419] [Test ami_limited_supervision/test.lst (12640 samples) in 363.179s (actual decoding time 0.0287s/sample) -- WER: 26.5974%, TER: 13.0674%]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0jcXZGQmm-Rs"
      },
      "source": [
        "We can see that the viterbi WER is 26.6% before finetuning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ryc17nPTds7Q"
      },
      "source": [
        "## Step 3: Run Finetuning\n",
        "\n",
        "\n",
        "Now, let's run finetuning with the AMI Corpus to see if we can improve the WER. \n",
        "\n",
        "Important parameters for `fl_asr_finetune_ctc`:\n",
        "\n",
        "`--train`, `--valid` - list files for training and validation sets respectively. Use comma to separate multiple files\n",
        "\n",
        "`--datadir` - [optional] base path to be used for `--train`, `--valid` flags\n",
        "\n",
        "`--lr` - learning rate for SGD\n",
        "\n",
        "`--momentum` - SGD momentum \n",
        "\n",
        "`--lr_decay` - epoch at which learning decay starts \n",
        "\n",
        "`--lr_decay_step` - learning rate halves after this epoch interval starting from epoch given by `lr_decay`  \n",
        "\n",
        "`--arch` - architecture file. Tune droupout if necessary. \n",
        "\n",
        "`--tokens` - tokens file \n",
        "\n",
        "`--batchsize` - batchsize per process\n",
        "\n",
        "`--lexicon` - lexicon file \n",
        "\n",
        "`--rundir` - path to store checkpoint logs\n",
        "\n",
        "`--reportiters` - Number of updates after which we will run evaluation on validation data and save model, if 0 we only do this at end of each epoch\n",
        "\n",
        "\n",
        ">Amount of train data | Config to use \n",
        ">---|---|\n",
        "> 10 min| --train train_10min_0.lst\n",
        "> 1 hr| --train train_10min_0.lst,train_10min_1.lst,train_10min_2.lst,train_10min_3.lst,train_10min_4.lst,train_10min_5.lst\n",
        "> 10 hr| --train train_10min_0.lst,train_10min_1.lst,train_10min_2.lst,train_10min_3.lst,train_10min_4.lst,train_10min_5.lst,train_9hr.lst\n",
        "\n",
        "Let's run finetuning with 10hr AMI data (**~7min** for 1000 updates with evaluation on dev set)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1RscueA4dlz0",
        "outputId": "4906a93a-28c4-4078-c750-1bbc39f4e3e6"
      },
      "source": [
        "! ./flashlight/build/bin/asr/fl_asr_tutorial_finetune_ctc model.bin \\\n",
        "  --datadir ami_limited_supervision \\\n",
        "  --train train_10min_0.lst,train_10min_1.lst,train_10min_2.lst,train_10min_3.lst,train_10min_4.lst,train_10min_5.lst,train_9hr.lst \\\n",
        "  --valid dev:dev.lst \\\n",
        "  --arch arch.txt \\\n",
        "  --tokens tokens.txt \\\n",
        "  --lexicon lexicon.txt \\\n",
        "  --rundir checkpoint \\\n",
        "  --lr 0.025 \\\n",
        "  --netoptim sgd \\\n",
        "  --momentum 0.8 \\\n",
        "  --reportiters 1000 \\\n",
        "  --lr_decay 100 \\\n",
        "  --lr_decay_step 50 \\\n",
        "  --iter 25000 \\\n",
        "  --batchsize 4 \\\n",
        "  --warmup 0"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I1224 06:39:48.599629 11517 FinetuneCTC.cpp:76] Parsing command line flags\n",
            "Initialized NCCL 2.7.8 successfully!\n",
            "I1224 06:39:49.002488 11517 FinetuneCTC.cpp:106] Gflags after parsing \n",
            "--flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --adambeta1=0.94999999999999996; --adambeta2=0.98999999999999999; --am=; --am_decoder_tr_dropout=0.20000000000000001; --am_decoder_tr_layerdrop=0.20000000000000001; --am_decoder_tr_layers=6; --arch=arch.txt; --attention=keyvalue; --attentionthreshold=2147483647; --attnWindow=softPretrain; --attnconvchannel=0; --attnconvkernel=0; --attndim=0; --batching_max_duration=0; --batching_strategy=none; --batchsize=4; --beamsize=2500; --beamsizetoken=250000; --beamthreshold=25; --channels=1; --criterion=ctc; --critoptim=adagrad; --datadir=ami_limited_supervision; --decoderattnround=1; --decoderdropout=0; --decoderrnnlayer=1; --decodertype=wrd; --devwin=0; --emission_dir=; --emission_queue_size=3000; --enable_distributed=true; --encoderdim=256; --eosscore=0; --eostoken=false; --everstoredb=false; --fftcachesize=1; --filterbanks=80; --fl_amp_max_scale_factor=32000; --fl_amp_scale_factor=4096; --fl_amp_scale_factor_update_interval=2000; --fl_amp_use_mixed_precision=false; --fl_benchmark_mode=true; --fl_log_level=; --fl_optim_mode=; --fl_vlog_level=0; --flagsfile=; --framesizems=25; --framestridems=10; --gamma=1; --gumbeltemperature=1; --highfreqfilterbank=-1; --inputfeeding=false; --isbeamdump=false; --iter=25000; --itersave=false; --labelsmooth=0.050000000000000003; --leftWindowSize=50; --lexicon=lexicon.txt; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lm_memory=5000; --lm_vocab=; --lmtype=kenlm; --lmweight=0; --lmweight_high=4; --lmweight_low=0; --lmweight_step=0.20000000000000001; --localnrmlleftctx=0; --localnrmlrightctx=0; --logadd=false; --lowfreqfilterbank=0; --lr=0.025000000000000001; --lr_decay=100; --lr_decay_step=50; --lrcosine=false; --lrcrit=0.02; --max_devices_per_node=8; --maxdecoderoutputlen=400; --maxgradnorm=0.10000000000000001; --maxload=-1; --maxrate=10; --maxsil=50; --maxword=-1; --melfloor=1; --mfcc=false; --mfcccoeffs=13; --mfsc=true; --minrate=3; --minsil=0; --momentum=0.80000000000000004; --netoptim=sgd; --nthread=6; --nthread_decoder=1; --nthread_decoder_am_forward=1; --numattnhead=8; --onorm=target; --optimepsilon=1e-08; --optimrho=0.90000000000000002; --pctteacherforcing=99; --pcttraineval=1; --pow=false; --pretrainWindow=0; --replabel=0; --reportiters=1000; --rightWindowSize=50; --rndv_filepath=; --rundir=checkpoint; --samplerate=16000; --sampletarget=0.01; --samplingstrategy=rand; --saug_fmaskf=30; --saug_fmaskn=2; --saug_start_update=24000; --saug_tmaskn=10; --saug_tmaskp=0.050000000000000003; --saug_tmaskt=30; --sclite=; --seed=0; --sfx_config=; --show=false; --showletters=false; --silscore=0; --smearing=none; --smoothingtemperature=1; --softwoffset=10; --softwrate=5; --softwstd=4; --sqnorm=true; --stepsize=9223372036854775807; --surround=; --test=; --tokens=tokens.txt; --train=train_10min_0.lst,train_10min_1.lst,train_10min_2.lst,train_10min_3.lst,train_10min_4.lst,train_10min_5.lst,train_9hr.lst; --trainWithWindow=true; --transdiag=0; --unkscore=-inf; --use_memcache=false; --uselexicon=true; --usewordpiece=false; --valid=dev:dev.lst; --validbatchsize=-1; --warmup=0; --weightdecay=0; --wordscore=0; --wordseparator=|; --world_rank=0; --world_size=1; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=; \n",
            "I1224 06:39:49.002910 11517 FinetuneCTC.cpp:107] Experiment path: checkpoint\n",
            "I1224 06:39:49.002919 11517 FinetuneCTC.cpp:108] Experiment runidx: 1\n",
            "I1224 06:39:49.003252 11517 FinetuneCTC.cpp:153] Number of classes (network): 29\n",
            "I1224 06:39:49.248888 11517 FinetuneCTC.cpp:160] Number of words: 200001\n",
            "I1224 06:39:50.344347 11517 FinetuneCTC.cpp:248] Loading architecture file from arch.txt\n",
            "I1224 06:39:50.868463 11517 FinetuneCTC.cpp:277] [Network] Sequential [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> (32) -> (33) -> (34) -> (35) -> (36) -> (37) -> (38) -> (39) -> (40) -> (41) -> (42) -> output]\n",
            "\t(0): View (-1 1 80 0)\n",
            "\t(1): LayerNorm ( axis : { 0 1 2 } , size : -1)\n",
            "\t(2): Conv2D (80->768, 7x1, 3,1, SAME,0, 1, 1) (with bias)\n",
            "\t(3): GatedLinearUnit (2)\n",
            "\t(4): Dropout (0.050000)\n",
            "\t(5): Reorder (2,0,3,1)\n",
            "\t(6): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(7): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(8): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(9): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(10): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(11): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(12): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(13): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(14): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(15): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(16): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(17): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(18): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(19): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(20): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(21): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(22): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(23): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(24): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(25): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(26): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(27): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(28): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(29): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(30): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(31): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(32): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(33): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(34): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(35): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(36): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(37): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(38): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(39): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(40): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(41): Transformer (nHeads: 4), (pDropout: 0.05), (pLayerdrop: 0.05), (bptt: 920), (useMask: 0), (preLayerNorm: 0)\n",
            "\t(42): Linear (384->29) (with bias)\n",
            "I1224 06:39:50.868662 11517 FinetuneCTC.cpp:278] [Network Params: 70498735]\n",
            "I1224 06:39:50.868705 11517 FinetuneCTC.cpp:283] [Criterion] ConnectionistTemporalClassificationCriterion\n",
            "I1224 06:39:51.004284 11517 FinetuneCTC.cpp:287] [Network Optimizer] SGD (momentum=0.8)\n",
            "I1224 06:39:51.005266 11517 FinetuneCTC.cpp:547] Shuffling trainset\n",
            "I1224 06:39:51.005805 11517 FinetuneCTC.cpp:554] Epoch 1 started!\n",
            "I1224 06:46:52.988443 11517 FinetuneCTC.cpp:331] epoch:        1 | nupdates:         1000 | lr: 0.025000 | lrcriterion: 0.000000 | runtime: 00:03:32 | bch(ms): 212.42 | smp(ms): 3.21 | fwd(ms): 82.29 | crit-fwd(ms): 2.90 | bwd(ms): 94.35 | optim(ms): 31.26 | loss:    2.78453 | train-TER:  9.68 | train-WER: 21.00 | dev-loss:    2.59365 | dev-TER: 10.09 | dev-WER: 19.50 | avg-isz: 287 | avg-tsz: 040 | max-tsz: 339 | avr-batchsz:    4.00 | hrs:    3.20 | thrpt(sec/sec): 54.18 | timestamp: 2020-12-24 06:46:52\n",
            "Memory Manager Stats\n",
            "Type: CachingMemoryManager\n",
            "Device: 0, Capacity: 14.72 GiB, Allocated: 12.90 GiB, Cached: 12.36 GiB\n",
            "Total native calls: 1059(mallocs), 541(frees)\n",
            "I1224 06:53:31.970283 11517 FinetuneCTC.cpp:331] epoch:        1 | nupdates:         2000 | lr: 0.025000 | lrcriterion: 0.000000 | runtime: 00:03:06 | bch(ms): 186.76 | smp(ms): 0.04 | fwd(ms): 69.67 | crit-fwd(ms): 2.02 | bwd(ms): 86.59 | optim(ms): 30.22 | loss:    2.63802 | train-TER:  9.86 | train-WER: 22.43 | dev-loss:    2.54714 | dev-TER:  9.84 | dev-WER: 18.86 | avg-isz: 259 | avg-tsz: 036 | max-tsz: 345 | avr-batchsz:    4.00 | hrs:    2.88 | thrpt(sec/sec): 55.57 | timestamp: 2020-12-24 06:53:31\n",
            "Memory Manager Stats\n",
            "Type: CachingMemoryManager\n",
            "Device: 0, Capacity: 14.72 GiB, Allocated: 12.90 GiB, Cached: 12.36 GiB\n",
            "Total native calls: 1059(mallocs), 541(frees)\n",
            "I1224 07:00:07.246326 11517 FinetuneCTC.cpp:331] epoch:        1 | nupdates:         3000 | lr: 0.025000 | lrcriterion: 0.000000 | runtime: 00:03:02 | bch(ms): 182.40 | smp(ms): 0.03 | fwd(ms): 67.86 | crit-fwd(ms): 1.92 | bwd(ms): 84.27 | optim(ms): 30.00 | loss:    2.57714 | train-TER: 12.22 | train-WER: 24.38 | dev-loss:    2.45296 | dev-TER:  9.55 | dev-WER: 18.37 | avg-isz: 248 | avg-tsz: 035 | max-tsz: 257 | avr-batchsz:    4.00 | hrs:    2.76 | thrpt(sec/sec): 54.53 | timestamp: 2020-12-24 07:00:07\n",
            "Memory Manager Stats\n",
            "Type: CachingMemoryManager\n",
            "Device: 0, Capacity: 14.72 GiB, Allocated: 12.90 GiB, Cached: 12.36 GiB\n",
            "Total native calls: 1059(mallocs), 541(frees)\n",
            "I1224 07:01:30.886020 11517 FinetuneCTC.cpp:547] Shuffling trainset\n",
            "I1224 07:01:30.886448 11517 FinetuneCTC.cpp:554] Epoch 2 started!\n",
            "[5e8e495af856:11519] *** Process received signal ***\n",
            "[5e8e495af856:11519] Signal: Segmentation fault (11)\n",
            "[5e8e495af856:11519] Signal code: Address not mapped (1)\n",
            "[5e8e495af856:11519] Failing at address: 0x7f848b62120d\n",
            "[5e8e495af856:11519] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f848e2cd980]\n",
            "[5e8e495af856:11519] [ 1] /lib/x86_64-linux-gnu/libc.so.6(getenv+0xa5)[0x7f848df0c8a5]\n",
            "[5e8e495af856:11519] [ 2] /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4(_ZN13TCMallocGuardD1Ev+0x34)[0x7f848e777e44]\n",
            "[5e8e495af856:11519] [ 3] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xf5)[0x7f848df0d735]\n",
            "[5e8e495af856:11519] [ 4] /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4(+0x13cb3)[0x7f848e775cb3]\n",
            "[5e8e495af856:11519] *** End of error message ***\n",
            "^C\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4BCed5Whw5YQ"
      },
      "source": [
        "## Step 4: Run Decoding "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e3pHr97mLp8W"
      },
      "source": [
        "#### Viterbi decoding\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OrrthfLYAfB_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d5cea038-9d0a-40ff-dcd2-fe998eec39d7"
      },
      "source": [
        "! ./flashlight/build/bin/asr/fl_asr_test --am checkpoint/001_model_dev.bin --datadir ''  --emission_dir '' --uselexicon false \\\n",
        "            --test ami_limited_supervision/test.lst --tokens tokens.txt --lexicon lexicon.txt --show "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[1;30;43mStreaming output truncated to the last 100 lines.\u001b[0m\n",
            "[sample: EN2002b_H02_FEO072_4.48_4.59, WER: 100%, TER: 100%, total WER: 19.4745%, total TER: 8.68993%, progress (thread 0): 99.7389%]\n",
            "|T|: y e p\n",
            "|P|: m\n",
            "[sample: EN2002b_H03_MEE073_307.97_308.08, WER: 100%, TER: 100%, total WER: 19.4754%, total TER: 8.69056%, progress (thread 0): 99.7468%]\n",
            "|T|: h m m\n",
            "|P|: m\n",
            "[sample: EN2002b_H03_MEE073_1578.82_1578.93, WER: 100%, TER: 66.6667%, total WER: 19.4763%, total TER: 8.69097%, progress (thread 0): 99.7547%]\n",
            "|T|: h m m\n",
            "|P|: m e\n",
            "[sample: EN2002c_H03_MEE073_1446.36_1446.47, WER: 100%, TER: 66.6667%, total WER: 19.4772%, total TER: 8.69137%, progress (thread 0): 99.7627%]\n",
            "|T|: o h\n",
            "|P|: h\n",
            "[sample: EN2002d_H01_FEO072_118.11_118.22, WER: 100%, TER: 50%, total WER: 19.4781%, total TER: 8.69156%, progress (thread 0): 99.7706%]\n",
            "|T|: s o\n",
            "|P|: m\n",
            "[sample: EN2002d_H02_MEE071_2107.05_2107.16, WER: 100%, TER: 100%, total WER: 19.479%, total TER: 8.69199%, progress (thread 0): 99.7785%]\n",
            "|T|: t\n",
            "|P|: m\n",
            "[sample: ES2004c_H02_MEE014_1184.81_1184.91, WER: 100%, TER: 100%, total WER: 19.4799%, total TER: 8.6922%, progress (thread 0): 99.7864%]\n",
            "|T|: y e s\n",
            "|P|: m a\n",
            "[sample: IS1009d_H01_FIO087_1744.56_1744.66, WER: 100%, TER: 100%, total WER: 19.4808%, total TER: 8.69284%, progress (thread 0): 99.7943%]\n",
            "|T|: o h\n",
            "|P|: h\n",
            "[sample: TS3003c_H02_MTD0010ID_1023.66_1023.76, WER: 100%, TER: 50%, total WER: 19.4817%, total TER: 8.69303%, progress (thread 0): 99.8022%]\n",
            "|T|: h m m\n",
            "|P|: m\n",
            "[sample: EN2002a_H00_MEE073_2013_2013.1, WER: 100%, TER: 66.6667%, total WER: 19.4826%, total TER: 8.69343%, progress (thread 0): 99.8101%]\n",
            "|T|: y e a h\n",
            "|P|: m\n",
            "[sample: EN2002b_H00_FEO070_43.63_43.73, WER: 100%, TER: 100%, total WER: 19.4835%, total TER: 8.69428%, progress (thread 0): 99.818%]\n",
            "|T|: y e a h\n",
            "|P|: y e a\n",
            "[sample: EN2002b_H03_MEE073_293.52_293.62, WER: 100%, TER: 25%, total WER: 19.4844%, total TER: 8.69443%, progress (thread 0): 99.826%]\n",
            "|T|: y e p\n",
            "|P|: y e\n",
            "[sample: EN2002c_H03_MEE073_763_763.1, WER: 100%, TER: 33.3333%, total WER: 19.4853%, total TER: 8.6946%, progress (thread 0): 99.8339%]\n",
            "|T|: m m\n",
            "|P|: m\n",
            "[sample: EN2002d_H03_MEE073_2099.51_2099.61, WER: 100%, TER: 50%, total WER: 19.4862%, total TER: 8.69479%, progress (thread 0): 99.8418%]\n",
            "|T|: ' k a y\n",
            "|P|: m\n",
            "[sample: ES2004a_H00_MEO015_168.79_168.88, WER: 100%, TER: 100%, total WER: 19.4871%, total TER: 8.69564%, progress (thread 0): 99.8497%]\n",
            "|T|: y e p\n",
            "|P|: y e\n",
            "[sample: ES2004c_H00_MEO015_360.73_360.82, WER: 100%, TER: 33.3333%, total WER: 19.488%, total TER: 8.69581%, progress (thread 0): 99.8576%]\n",
            "|T|: m m h m m\n",
            "|P|: m\n",
            "[sample: ES2004c_H03_FEE016_1132.2_1132.29, WER: 100%, TER: 80%, total WER: 19.4889%, total TER: 8.69664%, progress (thread 0): 99.8655%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: ES2004c_H03_FEE016_1651.84_1651.93, WER: 100%, TER: 100%, total WER: 19.4898%, total TER: 8.69728%, progress (thread 0): 99.8734%]\n",
            "|T|: m m h m m\n",
            "|P|: m\n",
            "[sample: IS1009c_H02_FIO084_1753.49_1753.58, WER: 100%, TER: 80%, total WER: 19.4907%, total TER: 8.69811%, progress (thread 0): 99.8813%]\n",
            "|T|: y e s\n",
            "|P|: \n",
            "[sample: IS1009d_H01_FIO087_749.88_749.97, WER: 100%, TER: 100%, total WER: 19.4916%, total TER: 8.69874%, progress (thread 0): 99.8892%]\n",
            "|T|: m m h m m\n",
            "|P|: \n",
            "[sample: IS1009d_H02_FIO084_1718.32_1718.41, WER: 100%, TER: 100%, total WER: 19.4925%, total TER: 8.6998%, progress (thread 0): 99.8972%]\n",
            "|T|: w h a t\n",
            "|P|: m\n",
            "[sample: TS3003d_H00_MTD009PM_1597.33_1597.42, WER: 100%, TER: 100%, total WER: 19.4934%, total TER: 8.70065%, progress (thread 0): 99.9051%]\n",
            "|T|: h m m\n",
            "|P|: m\n",
            "[sample: EN2002a_H00_MEE073_887.36_887.45, WER: 100%, TER: 66.6667%, total WER: 19.4943%, total TER: 8.70105%, progress (thread 0): 99.913%]\n",
            "|T|: d a m n\n",
            "|P|: m\n",
            "[sample: EN2002a_H00_MEE073_1500.91_1501, WER: 100%, TER: 75%, total WER: 19.4952%, total TER: 8.70167%, progress (thread 0): 99.9209%]\n",
            "|T|: h m m\n",
            "|P|: m\n",
            "[sample: EN2002c_H03_MEE073_429.63_429.72, WER: 100%, TER: 66.6667%, total WER: 19.4961%, total TER: 8.70207%, progress (thread 0): 99.9288%]\n",
            "|T|: y e a h\n",
            "|P|: y a\n",
            "[sample: EN2002c_H02_MEE071_2578.94_2579.03, WER: 100%, TER: 50%, total WER: 19.497%, total TER: 8.70246%, progress (thread 0): 99.9367%]\n",
            "|T|: d a m n\n",
            "|P|: y e a\n",
            "[sample: EN2002d_H03_MEE073_999.94_1000.03, WER: 100%, TER: 100%, total WER: 19.4979%, total TER: 8.7033%, progress (thread 0): 99.9446%]\n",
            "|T|: m m\n",
            "|P|: \n",
            "[sample: ES2004a_H00_MEO015_252.48_252.56, WER: 100%, TER: 100%, total WER: 19.4988%, total TER: 8.70373%, progress (thread 0): 99.9525%]\n",
            "|T|: m m\n",
            "|P|: \n",
            "[sample: ES2004c_H00_MEO015_1885.03_1885.11, WER: 100%, TER: 100%, total WER: 19.4997%, total TER: 8.70415%, progress (thread 0): 99.9604%]\n",
            "|T|: o h\n",
            "|P|: \n",
            "[sample: TS3003a_H01_MTD011UID_1313.86_1313.94, WER: 100%, TER: 100%, total WER: 19.5006%, total TER: 8.70458%, progress (thread 0): 99.9684%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: EN2002a_H00_MEE073_185.9_185.98, WER: 100%, TER: 100%, total WER: 19.5014%, total TER: 8.70521%, progress (thread 0): 99.9763%]\n",
            "|T|: g\n",
            "|P|: m\n",
            "[sample: EN2002d_H03_MEE073_241.62_241.69, WER: 100%, TER: 100%, total WER: 19.5023%, total TER: 8.70542%, progress (thread 0): 99.9842%]\n",
            "|T|: h m m\n",
            "|P|: \n",
            "[sample: IS1009a_H02_FIO084_490.4_490.46, WER: 100%, TER: 100%, total WER: 19.5032%, total TER: 8.70606%, progress (thread 0): 99.9921%]\n",
            "|T|: ' k a y\n",
            "|P|: \n",
            "[sample: EN2002b_H03_MEE073_613.94_614, WER: 100%, TER: 100%, total WER: 19.5041%, total TER: 8.70691%, progress (thread 0): 100%]\n",
            "I1224 07:09:02.842401 11830 Test.cpp:418] ------\n",
            "I1224 07:09:02.842427 11830 Test.cpp:419] [Test ami_limited_supervision/test.lst (12640 samples) in 361.478s (actual decoding time 0.0286s/sample) -- WER: 19.5041%, TER: 8.70691%]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6oz870UrAkJu"
      },
      "source": [
        "Viterbi WER improved from 26.6% to 19.5% after 1 epoch with finetuning..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3iJXpsQLwW0"
      },
      "source": [
        "#### Beam Search decoding with a language model \n",
        "\n",
        "To do this, download the finetuned model and use the [Inference CTC tutorial](https://colab.research.google.com/github/flashlight/flashlight/blob/master/flashlight/app/asr/tutorial/notebooks/InferenceAndAlignmentCTC.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPmZlgbRNmh1"
      },
      "source": [
        "## Step 5: Running with your own data \n",
        "\n",
        "To finetune on your own data, create `train`, `dev` and `test` list files and run the finetuning step. \n",
        "\n",
        "Each list file consists of multiple lines with each line describing one sample in the following format : \n",
        "```\n",
        "<sample_id> <path_to_audio_file> <duration> <transcript>\n",
        "```\n",
        "\n",
        "For example, let's take a look at the `dev.lst` file from AMI corpus."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MKW6ktM7NxWV",
        "outputId": "19020539-f010-4670-cd81-60fa92f79593"
      },
      "source": [
        "! head ami_limited_supervision/dev.lst "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ES2011a_H00_FEE041_34.27_37.14\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_34.27_37.14.flac\t2870.0\there we go\n",
            "ES2011a_H00_FEE041_37.14_39.15\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_37.14_39.15.flac\t2010.0\twelcome everybody\n",
            "ES2011a_H00_FEE041_43.32_44.39\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_43.32_44.39.flac\t1070.0\tyou can call me abbie\n",
            "ES2011a_H00_FEE041_39.15_43.32\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_39.15_43.32.flac\t4170.0\tum i'm abigail claflin\n",
            "ES2011a_H00_FEE041_46.43_47.63\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_46.43_47.63.flac\t1200.0\t's see\n",
            "ES2011a_H00_FEE041_51.33_55.53\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_51.33_55.53.flac\t4200.0\tso this is our kick off meeting\n",
            "ES2011a_H00_FEE041_55.53_56.85\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_55.53_56.85.flac\t1320.0\tum\n",
            "ES2011a_H00_FEE041_47.63_50.2\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_47.63_50.2.flac\t2570.0\tpowerpoint that's not it\n",
            "ES2011a_H00_FEE041_50.2_51.33\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_50.2_51.33.flac\t1130.0\tthere we go\n",
            "ES2011a_H00_FEE041_62.17_64.28\tami_limited_supervision/audio/ES2011a/ES2011a_H00_FEE041_62.17_64.28.flac\t2110.0\tlet's shall we all introduce ourselves\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Ag_BKtdPedy"
      },
      "source": [
        "#### Recording your own audio\n",
        "\n",
        "For example, you can record your own audio and finetune the model...\n",
        "\n",
        "Installing a few packages first..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z9XKBvzpPqm4"
      },
      "source": [
        "!apt-get install sox\n",
        "!pip install ffmpeg-python sox"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LpPIM81ZLGnn"
      },
      "source": [
        "from flashlight.scripts.colab.record import record_audio"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WaxI4DAaK6gM"
      },
      "source": [
        "**Let's record now the following sentences:**\n",
        "\n",
        "**1:** A flashlight or torch is a small, portable spotlight. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mEFlB90VOtaF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "a9b1f261-117f-4ffb-9212-1ab022d2e8e2"
      },
      "source": [
        "record_audio(\"recorded_audio_1\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "<script>\n",
              "var recordButton = document.createElement(\"BUTTON\");\n",
              "recordButton.appendChild(\n",
              "  document.createTextNode(\"Press to start recording\")\n",
              ");\n",
              "restyleButtonBeforeRecording();\n",
              "\n",
              "var my_div = document.createElement(\"DIV\");\n",
              "my_div.appendChild(recordButton);\n",
              "\n",
              "document.body.appendChild(my_div);\n",
              "\n",
              "var base64data = 0;\n",
              "var reader;\n",
              "var recorder, gumStream;\n",
              "\n",
              "function restyleButtonBeforeRecording() {\n",
              "  recordButton.style.width = '270px';\n",
              "  recordButton.style.height = '90';\n",
              "  recordButton.style.padding = '25px';\n",
              "  recordButton.style.backgroundColor = '#4CAF50';\n",
              "  recordButton.style.fontSize = '18px';\n",
              "}\n",
              "\n",
              "function restyleButtonForRecording() {\n",
              "  recordButton.style.backgroundColor = '#008CBA';\n",
              "  recordButton.innerText = \"Recording... press to stop\";\n",
              "}\n",
              "\n",
              "function restyleButtonForSaving() {\n",
              "  recordButton.style.backgroundColor = '#b34d4d';\n",
              "  recordButton.innerText = \"Saving... please wait!\"\n",
              "}\n",
              "\n",
              "var handleSuccess = function(stream) {\n",
              "  gumStream = stream;\n",
              "  recorder = new MediaRecorder(stream);\n",
              "  recorder.ondataavailable = function(e) {\n",
              "    var url = URL.createObjectURL(e.data);\n",
              "    var preview = document.createElement('audio');\n",
              "    preview.controls = true;\n",
              "    preview.src = url;\n",
              "    document.body.appendChild(preview);\n",
              "\n",
              "    reader = new FileReader();\n",
              "    reader.readAsDataURL(e.data);\n",
              "    reader.onloadend = function() {\n",
              "      base64data = reader.result;\n",
              "      //console.log(\"Inside FileReader:\" + base64data);\n",
              "    }\n",
              "  };\n",
              "  recorder.start();\n",
              "  };\n",
              "\n",
              "\n",
              "function toggleRecording() {\n",
              "  if (recorder && recorder.state == \"recording\") {\n",
              "      recorder.stop();\n",
              "      gumStream.getAudioTracks()[0].stop();\n",
              "      restyleButtonForSaving();\n",
              "  }\n",
              "}\n",
              "\n",
              "// https://stackoverflow.com/a/951057\n",
              "function sleep(ms) {\n",
              "  return new Promise(resolve => setTimeout(resolve, ms));\n",
              "}\n",
              "\n",
              "var data = new Promise(resolve=>{\n",
              "  recordButton.onclick = () => {\n",
              "    restyleButtonForRecording();\n",
              "    recordButton.onclick = () => {\n",
              "      toggleRecording();\n",
              "      sleep(2000).then(() => {\n",
              "        // wait 2000ms for the data to be available...\n",
              "        // ideally this should use something like await...\n",
              "        // console.log(\"Inside data:\" + base64data)\n",
              "        resolve(base64data.toString());\n",
              "      });\n",
              "    };\n",
              "    navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
              "  };\n",
              "});\n",
              "\n",
              "</script>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0DEuxuwLLOc"
      },
      "source": [
        "**2:** Its function is a beam of light which helps to see and it usually requires batteries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "id": "N84pjDEsKs6S",
        "outputId": "f546e80e-671f-43c6-8822-0db9f7271946"
      },
      "source": [
        "record_audio(\"recorded_audio_2\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "<script>\n",
              "var recordButton = document.createElement(\"BUTTON\");\n",
              "recordButton.appendChild(\n",
              "  document.createTextNode(\"Press to start recording\")\n",
              ");\n",
              "restyleButtonBeforeRecording();\n",
              "\n",
              "var my_div = document.createElement(\"DIV\");\n",
              "my_div.appendChild(recordButton);\n",
              "\n",
              "document.body.appendChild(my_div);\n",
              "\n",
              "var base64data = 0;\n",
              "var reader;\n",
              "var recorder, gumStream;\n",
              "\n",
              "function restyleButtonBeforeRecording() {\n",
              "  recordButton.style.width = '270px';\n",
              "  recordButton.style.height = '90';\n",
              "  recordButton.style.padding = '25px';\n",
              "  recordButton.style.backgroundColor = '#4CAF50';\n",
              "  recordButton.style.fontSize = '18px';\n",
              "}\n",
              "\n",
              "function restyleButtonForRecording() {\n",
              "  recordButton.style.backgroundColor = '#008CBA';\n",
              "  recordButton.innerText = \"Recording... press to stop\";\n",
              "}\n",
              "\n",
              "function restyleButtonForSaving() {\n",
              "  recordButton.style.backgroundColor = '#b34d4d';\n",
              "  recordButton.innerText = \"Saving... please wait!\"\n",
              "}\n",
              "\n",
              "var handleSuccess = function(stream) {\n",
              "  gumStream = stream;\n",
              "  recorder = new MediaRecorder(stream);\n",
              "  recorder.ondataavailable = function(e) {\n",
              "    var url = URL.createObjectURL(e.data);\n",
              "    var preview = document.createElement('audio');\n",
              "    preview.controls = true;\n",
              "    preview.src = url;\n",
              "    document.body.appendChild(preview);\n",
              "\n",
              "    reader = new FileReader();\n",
              "    reader.readAsDataURL(e.data);\n",
              "    reader.onloadend = function() {\n",
              "      base64data = reader.result;\n",
              "      //console.log(\"Inside FileReader:\" + base64data);\n",
              "    }\n",
              "  };\n",
              "  recorder.start();\n",
              "  };\n",
              "\n",
              "\n",
              "function toggleRecording() {\n",
              "  if (recorder && recorder.state == \"recording\") {\n",
              "      recorder.stop();\n",
              "      gumStream.getAudioTracks()[0].stop();\n",
              "      restyleButtonForSaving();\n",
              "  }\n",
              "}\n",
              "\n",
              "// https://stackoverflow.com/a/951057\n",
              "function sleep(ms) {\n",
              "  return new Promise(resolve => setTimeout(resolve, ms));\n",
              "}\n",
              "\n",
              "var data = new Promise(resolve=>{\n",
              "  recordButton.onclick = () => {\n",
              "    restyleButtonForRecording();\n",
              "    recordButton.onclick = () => {\n",
              "      toggleRecording();\n",
              "      sleep(2000).then(() => {\n",
              "        // wait 2000ms for the data to be available...\n",
              "        // ideally this should use something like await...\n",
              "        // console.log(\"Inside data:\" + base64data)\n",
              "        resolve(base64data.toString());\n",
              "      });\n",
              "    };\n",
              "    navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
              "  };\n",
              "});\n",
              "\n",
              "</script>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fIrr7DrLb_L"
      },
      "source": [
        "**3:** In 1896, the first dry cell battery was invented. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "id": "qGn8e_cALcZO",
        "outputId": "4accd936-7a4d-4fcb-a3ee-dd30ee5823f1"
      },
      "source": [
        "record_audio(\"recorded_audio_3\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "<script>\n",
              "var recordButton = document.createElement(\"BUTTON\");\n",
              "recordButton.appendChild(\n",
              "  document.createTextNode(\"Press to start recording\")\n",
              ");\n",
              "restyleButtonBeforeRecording();\n",
              "\n",
              "var my_div = document.createElement(\"DIV\");\n",
              "my_div.appendChild(recordButton);\n",
              "\n",
              "document.body.appendChild(my_div);\n",
              "\n",
              "var base64data = 0;\n",
              "var reader;\n",
              "var recorder, gumStream;\n",
              "\n",
              "function restyleButtonBeforeRecording() {\n",
              "  recordButton.style.width = '270px';\n",
              "  recordButton.style.height = '90';\n",
              "  recordButton.style.padding = '25px';\n",
              "  recordButton.style.backgroundColor = '#4CAF50';\n",
              "  recordButton.style.fontSize = '18px';\n",
              "}\n",
              "\n",
              "function restyleButtonForRecording() {\n",
              "  recordButton.style.backgroundColor = '#008CBA';\n",
              "  recordButton.innerText = \"Recording... press to stop\";\n",
              "}\n",
              "\n",
              "function restyleButtonForSaving() {\n",
              "  recordButton.style.backgroundColor = '#b34d4d';\n",
              "  recordButton.innerText = \"Saving... please wait!\"\n",
              "}\n",
              "\n",
              "var handleSuccess = function(stream) {\n",
              "  gumStream = stream;\n",
              "  recorder = new MediaRecorder(stream);\n",
              "  recorder.ondataavailable = function(e) {\n",
              "    var url = URL.createObjectURL(e.data);\n",
              "    var preview = document.createElement('audio');\n",
              "    preview.controls = true;\n",
              "    preview.src = url;\n",
              "    document.body.appendChild(preview);\n",
              "\n",
              "    reader = new FileReader();\n",
              "    reader.readAsDataURL(e.data);\n",
              "    reader.onloadend = function() {\n",
              "      base64data = reader.result;\n",
              "      //console.log(\"Inside FileReader:\" + base64data);\n",
              "    }\n",
              "  };\n",
              "  recorder.start();\n",
              "  };\n",
              "\n",
              "\n",
              "function toggleRecording() {\n",
              "  if (recorder && recorder.state == \"recording\") {\n",
              "      recorder.stop();\n",
              "      gumStream.getAudioTracks()[0].stop();\n",
              "      restyleButtonForSaving();\n",
              "  }\n",
              "}\n",
              "\n",
              "// https://stackoverflow.com/a/951057\n",
              "function sleep(ms) {\n",
              "  return new Promise(resolve => setTimeout(resolve, ms));\n",
              "}\n",
              "\n",
              "var data = new Promise(resolve=>{\n",
              "  recordButton.onclick = () => {\n",
              "    restyleButtonForRecording();\n",
              "    recordButton.onclick = () => {\n",
              "      toggleRecording();\n",
              "      sleep(2000).then(() => {\n",
              "        // wait 2000ms for the data to be available...\n",
              "        // ideally this should use something like await...\n",
              "        // console.log(\"Inside data:\" + base64data)\n",
              "        resolve(base64data.toString());\n",
              "      });\n",
              "    };\n",
              "    navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
              "  };\n",
              "});\n",
              "\n",
              "</script>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PLh3W-8LPTo"
      },
      "source": [
        "**4:** Unlike previous batteries, it used a paste electrolyte instead of a liquid."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "id": "TF813JDeKt6i",
        "outputId": "586b8f61-21a7-418d-bf21-802eeea3f0b1"
      },
      "source": [
        "record_audio(\"recorded_audio_4\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "<script>\n",
              "var recordButton = document.createElement(\"BUTTON\");\n",
              "recordButton.appendChild(\n",
              "  document.createTextNode(\"Press to start recording\")\n",
              ");\n",
              "restyleButtonBeforeRecording();\n",
              "\n",
              "var my_div = document.createElement(\"DIV\");\n",
              "my_div.appendChild(recordButton);\n",
              "\n",
              "document.body.appendChild(my_div);\n",
              "\n",
              "var base64data = 0;\n",
              "var reader;\n",
              "var recorder, gumStream;\n",
              "\n",
              "function restyleButtonBeforeRecording() {\n",
              "  recordButton.style.width = '270px';\n",
              "  recordButton.style.height = '90';\n",
              "  recordButton.style.padding = '25px';\n",
              "  recordButton.style.backgroundColor = '#4CAF50';\n",
              "  recordButton.style.fontSize = '18px';\n",
              "}\n",
              "\n",
              "function restyleButtonForRecording() {\n",
              "  recordButton.style.backgroundColor = '#008CBA';\n",
              "  recordButton.innerText = \"Recording... press to stop\";\n",
              "}\n",
              "\n",
              "function restyleButtonForSaving() {\n",
              "  recordButton.style.backgroundColor = '#b34d4d';\n",
              "  recordButton.innerText = \"Saving... please wait!\"\n",
              "}\n",
              "\n",
              "var handleSuccess = function(stream) {\n",
              "  gumStream = stream;\n",
              "  recorder = new MediaRecorder(stream);\n",
              "  recorder.ondataavailable = function(e) {\n",
              "    var url = URL.createObjectURL(e.data);\n",
              "    var preview = document.createElement('audio');\n",
              "    preview.controls = true;\n",
              "    preview.src = url;\n",
              "    document.body.appendChild(preview);\n",
              "\n",
              "    reader = new FileReader();\n",
              "    reader.readAsDataURL(e.data);\n",
              "    reader.onloadend = function() {\n",
              "      base64data = reader.result;\n",
              "      //console.log(\"Inside FileReader:\" + base64data);\n",
              "    }\n",
              "  };\n",
              "  recorder.start();\n",
              "  };\n",
              "\n",
              "\n",
              "function toggleRecording() {\n",
              "  if (recorder && recorder.state == \"recording\") {\n",
              "      recorder.stop();\n",
              "      gumStream.getAudioTracks()[0].stop();\n",
              "      restyleButtonForSaving();\n",
              "  }\n",
              "}\n",
              "\n",
              "// https://stackoverflow.com/a/951057\n",
              "function sleep(ms) {\n",
              "  return new Promise(resolve => setTimeout(resolve, ms));\n",
              "}\n",
              "\n",
              "var data = new Promise(resolve=>{\n",
              "  recordButton.onclick = () => {\n",
              "    restyleButtonForRecording();\n",
              "    recordButton.onclick = () => {\n",
              "      toggleRecording();\n",
              "      sleep(2000).then(() => {\n",
              "        // wait 2000ms for the data to be available...\n",
              "        // ideally this should use something like await...\n",
              "        // console.log(\"Inside data:\" + base64data)\n",
              "        resolve(base64data.toString());\n",
              "      });\n",
              "    };\n",
              "    navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
              "  };\n",
              "});\n",
              "\n",
              "</script>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sl3xY-w_LWpF"
      },
      "source": [
        "**5** This was the first battery suitable for portable electrical devices, as it did not spill or break easily and worked in any orientation. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "id": "Zpp6K6JJLZDW",
        "outputId": "4e953fe0-6f72-4589-89e6-2e71ac47a833"
      },
      "source": [
        "record_audio(\"recorded_audio_5\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "<script>\n",
              "var recordButton = document.createElement(\"BUTTON\");\n",
              "recordButton.appendChild(\n",
              "  document.createTextNode(\"Press to start recording\")\n",
              ");\n",
              "restyleButtonBeforeRecording();\n",
              "\n",
              "var my_div = document.createElement(\"DIV\");\n",
              "my_div.appendChild(recordButton);\n",
              "\n",
              "document.body.appendChild(my_div);\n",
              "\n",
              "var base64data = 0;\n",
              "var reader;\n",
              "var recorder, gumStream;\n",
              "\n",
              "function restyleButtonBeforeRecording() {\n",
              "  recordButton.style.width = '270px';\n",
              "  recordButton.style.height = '90';\n",
              "  recordButton.style.padding = '25px';\n",
              "  recordButton.style.backgroundColor = '#4CAF50';\n",
              "  recordButton.style.fontSize = '18px';\n",
              "}\n",
              "\n",
              "function restyleButtonForRecording() {\n",
              "  recordButton.style.backgroundColor = '#008CBA';\n",
              "  recordButton.innerText = \"Recording... press to stop\";\n",
              "}\n",
              "\n",
              "function restyleButtonForSaving() {\n",
              "  recordButton.style.backgroundColor = '#b34d4d';\n",
              "  recordButton.innerText = \"Saving... please wait!\"\n",
              "}\n",
              "\n",
              "var handleSuccess = function(stream) {\n",
              "  gumStream = stream;\n",
              "  recorder = new MediaRecorder(stream);\n",
              "  recorder.ondataavailable = function(e) {\n",
              "    var url = URL.createObjectURL(e.data);\n",
              "    var preview = document.createElement('audio');\n",
              "    preview.controls = true;\n",
              "    preview.src = url;\n",
              "    document.body.appendChild(preview);\n",
              "\n",
              "    reader = new FileReader();\n",
              "    reader.readAsDataURL(e.data);\n",
              "    reader.onloadend = function() {\n",
              "      base64data = reader.result;\n",
              "      //console.log(\"Inside FileReader:\" + base64data);\n",
              "    }\n",
              "  };\n",
              "  recorder.start();\n",
              "  };\n",
              "\n",
              "\n",
              "function toggleRecording() {\n",
              "  if (recorder && recorder.state == \"recording\") {\n",
              "      recorder.stop();\n",
              "      gumStream.getAudioTracks()[0].stop();\n",
              "      restyleButtonForSaving();\n",
              "  }\n",
              "}\n",
              "\n",
              "// https://stackoverflow.com/a/951057\n",
              "function sleep(ms) {\n",
              "  return new Promise(resolve => setTimeout(resolve, ms));\n",
              "}\n",
              "\n",
              "var data = new Promise(resolve=>{\n",
              "  recordButton.onclick = () => {\n",
              "    restyleButtonForRecording();\n",
              "    recordButton.onclick = () => {\n",
              "      toggleRecording();\n",
              "      sleep(2000).then(() => {\n",
              "        // wait 2000ms for the data to be available...\n",
              "        // ideally this should use something like await...\n",
              "        // console.log(\"Inside data:\" + base64data)\n",
              "        resolve(base64data.toString());\n",
              "      });\n",
              "    };\n",
              "    navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
              "  };\n",
              "});\n",
              "\n",
              "</script>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyfYgfHnLqxf"
      },
      "source": [
        "### Create now new training/dev lists:\n",
        "\n",
        "(yes, you need to edit transcriptions below to your recordings)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vLK6F1cyLtGp"
      },
      "source": [
        "import sox\n",
        "transcriptions = [\n",
        "   \"a flashlight or torch is a small portable spotlight\",\n",
        "   \"its function is a beam of light which helps to see and it usually requires batteries\",\n",
        "   \"in eighteen ninthy six the first dry cell battery was invented\",\n",
        "   \"unlike previous batteries it used a paste electrolyte instead of a liquid\",\n",
        "   \"this was the first battery suitable for portable electrical devices, as it did not spill or break easily and worked in any orientation\"\n",
        "]\n",
        "with open(\"own_train.lst\", \"w\") as f_train, open(\"own_dev.lst\", \"w\") as f_dev:\n",
        "  for index, transcription in enumerate(transcriptions):\n",
        "    fname = \"recorded_audio_\" + str(index + 1) + \".wav\"\n",
        "    duration_ms = sox.file_info.duration(fname) * 1000\n",
        "    if index % 2 == 0:\n",
        "      f_train.write(\"{}\\t{}\\t{}\\t{}\\n\".format(\n",
        "          index + 1, fname, duration_ms, transcription))\n",
        "    else:\n",
        "      f_dev.write(\"{}\\t{}\\t{}\\t{}\\n\".format(\n",
        "          index + 1, fname, duration_ms, transcription))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y8ou2LnDQONk"
      },
      "source": [
        "### Check at first model quality on dev before finetuning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fq4g9tCAQFw8"
      },
      "source": [
        "! ./flashlight/build/bin/asr/fl_asr_test --am model.bin --datadir ''  --emission_dir '' --uselexicon false \\\n",
        "            --test own_dev.lst --tokens tokens.txt --lexicon lexicon.txt --show "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VUy16oZRQjT1"
      },
      "source": [
        "### Finetune on recorded audio samples\n",
        "\n",
        "Play with parameters if needed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YAtnwKDXMtZ2"
      },
      "source": [
        "! ./flashlight/build/bin/asr/fl_asr_tutorial_finetune_ctc model.bin \\\n",
        "  --datadir= \\\n",
        "  --train own_train.lst \\\n",
        "  --valid dev:own_dev.lst \\\n",
        "  --arch arch.txt \\\n",
        "  --tokens tokens.txt \\\n",
        "  --lexicon lexicon.txt \\\n",
        "  --rundir own_checkpoint \\\n",
        "  --lr 0.025 \\\n",
        "  --netoptim sgd \\\n",
        "  --momentum 0.8 \\\n",
        "  --reportiters 1000 \\\n",
        "  --lr_decay 100 \\\n",
        "  --lr_decay_step 50 \\\n",
        "  --iter 25000 \\\n",
        "  --batchsize 4 \\\n",
        "  --warmup 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3dWOwSVgQ3bL"
      },
      "source": [
        "### Test finetuned model\n",
        "\n",
        "(unlikely you get significant improvement with just five phrases, but let's check!)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j4uG8XYVTqQu"
      },
      "source": [
        "! ./flashlight/build/bin/asr/fl_asr_test --am own_checkpoint/001_model_dev.bin --datadir ''  --emission_dir '' --uselexicon false \\\n",
        "            --test own_dev.lst --tokens tokens.txt --lexicon lexicon.txt --show "
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}
